Apparatus for recording the retina reflex image and for superimposing of additional images in the eye

ABSTRACT

An apparatus for recording the exterior retina reflex image of the human eye, has a scanning recorder. The beams of the exterior images pass through the spectacle glasses constructed on the inner side as vaulted, imaging, beam splitting mirrors. The scanning recorder for recording and detouring the parallel beam bundles exiting from the eye after backscattering by each spot on the retina, is a biaxial scanner. An opto-electronic detector receives the light bundles from the biaxial scanner for serial recording of the retina reflex image.

FIELD OF THE INVENTION

The invention relates to an apparatus which makes it possible to recordthe back scatter of the exterior image projected onto the retina of thehuman eye. The backscatter or retina reflex image is modified orenhanced by additional information by an electronic image processing toform an enhanced image which is projected back into the eye with the aidof a laser beam modulation and deflection for superimposing on theoriginal exterior image.

BACKGROUND INFORMATION

The technical requirements that must be met by the recording, processingand reproduction of images keep growing with an increasing demand forinformation and their clear graphic visualization. The rapid advance inthese fields goes hand-in-hand with the ever accelerating imageprocessing by computers.

Today, the main field where electronic image processing is applied,involves the further processing of images that are taken by cameras,scanning systems and sensors in the visible light spectrum as well as inother sections of the electromagnetic spectrum such as the infrared, theradio frequency and the X-ray frequency range. After electronicprocessing, the images are reproduced either as individual images or asmoving images on an image reproduction screen such as a display forpresenting the information to the eye.

On the one hand it is possible to make special image contents easierrecognizable by electronic image processing. Known techniques for thispurpose include, for example local frequency filtering, margin sharpnessincreasing, image data compression, image correlation, dynamicreductions and false color coding. On the other hand, other techniquesare concerned with the superposition or subtraction of auxiliary imagestaken from different spectral ranges or with the superimposing of storedplans, maps, and drawings onto the original image.

For many applications an image presentation practically free of time lagis of great advantage to the eye, for example when operating anaircraft, ship, vehicle, or in an open loop control and monitoring ofprocesses and assembly lines. By applying image processing theinformation content of the actual, direct image can be intentionallyincreased or reduced. Image processing is used in a wide range fromincreasing the image contrast to blending-in of additional information,marking of details, and highlighting dangers.

In many of these applications, it is disadvantageous that the electroniccamera is a “second eye system” separate from the human eye. Thisdisadvantage is due to the fact that the images are seen from anotherrecording location and that additionally, the pictures on the imagescreen are presented at another observation location than the eye. Thus,the human eye must constantly change between the direct observation andthe indirect observation while taking into account different observationangles, different image details, and different size ratios which leadsto physical impairments and delays when decisions must be made.

The above problems have in part been solved by the “head-up-display(HUD)” technique used in the piloting of combat aircraft, in thatimportant informations such as instrument displays and target data areinserted or fade-in into the open spectacles of the pilot's helmet andthus into the visual field of the pilot. This technique is also usedexperimentally in the automobile industry for displaying of instrumentreadings on the windshield so that the driver is not distracted fromviewing the road by viewing the instrument panel.

The HUD technique has been further developed in the so-called “virtualreality” or “cyber space” technique, wherein closed spectacles are used,i.e. glasses in which the outside view is blocked, and three-dimensionalfull images moved by the HUD, are projected into the eye with virtualreality. These virtual reality images are then modified in response tobody motions such as locomotion, movement of an arm, a finger, or headand eye movements.

The HUD technique generates an image on an image screen and projects theimage into the eye after reflection on the surface of the spectacleglasses. The eye sees, so to speak, through the glasses as full mirrorsonto the display “around the corner”. Where open spectacles are used, apartially transmitting mirror permits the simultaneous viewing of theoutside environment. Since the display is connected to the head, theimage follows the head movements.

Certain HUD devices are equipped with an “eye tracker” which follows theeye movements with the help of a motion sensor applied to the eyeball orwith a camera which observes the movements of the eye pupils or of thevascular structure of the retina. It is thus possible to electronicallyshift the image projected in the HUD device, corresponding to the thesemovements within the visual field.

It is possible in a HUD device to project the image through theprojection optic into “infinity” in order to relax the eye free ofaccommodation. By adjusting different view angles for both eyes towardthe same object, a stereoscopic, i.e. three-dimensional vision, ispossible.

On the one hand these applications and techniques illustrate the highlevel of the electronic image processing which is capable to processmoving images with an acceptable quality almost without time lag andwith a reasonable technical effort and expense. On the other hand, thesetechniques also illustrate the increasing demand for a direct imagetransmission into the eye.

However, there are limits to current HUD techniques. The accuracy orprecision of the automatic tracking of the eye movements with the eyetracker is substantially worse than the alignment precision and imageresolution of the eye. As a result, the fade-in image floats or dancesin the visual field which leads to an unprecise target acquisition andis tiring to the eyes.

For the above reasons, conventional applications of the full imagereproduction are limited to the use of closed spectacles, i.e. to theexclusive fade-in of external images. Contrary thereto, when openspectacles are used, permitting an additional external view, the fade-inis still limited to simple additional information in the form of text,symbols, or image contours.

A complete three-dimensional and timely overlap between fade-in imagesand the real image seen by the eye requires an exact three-dimensionaland timely coincidence of the two images on the retina. It is the aim ofthe invention to achieve this coincidence by a direct recording orphotographing of the retina image and then projecting the new image backonto the real image substantially without any time lag and withcongruence.

First, the prior art will be discussed as far as it relates to therecording of retina reflex images, to image scanning in the internal eyeand the projection of laser images directly into the eye. The inventionstarts from this prior art.

The technical realization of a continuous imaging of the retina reflexof the environment or exterior requires that the optical reflex of theretina is actually usable. The reflection capability of the retina hasbeen measured in detail, for example by F. C. Delori and K. P. Pflibsenin an article entitled “Spectral Reflectance of the Human Ocular Fundus”which appeared in “Applied Optics”, Vol. 28, No. 6, 1989. The reflectioncapability of the fovea centralis of the retina has a low value of 0.2%at the blue visual spectral range (450 nm) and increases monotonously toa value of 10% at the long wave red range (750 nm). In the range of thelargest eye sensitivity and the most acute vision, namely in thegreen/yellow range between 500 nm and 600 nm the reflection capabilityis within 1 and 2%.

Thus, a recording system for this reflection capability must beconstructed for an illumination density of the retina that is smaller bya factor of 50 to 100 compared to the illumination density of theobjects seen by the eye. A further impairment of the available lightquantity is due to the size of the eye pupil of 1 to 7 mm, which is,compared to conventional technical recording systems such asphotographic and video cameras, relatively small. The recording of thelight reflected by the retina thus requires, due to these two reasons,an especially sensitive light sensor.

It is known that a structured reflex image is generated in the area ofthe fovea centralis of the retina when an image is formed in the eye.This phenomenon is described, for example, by F. W. Campbell and D. G.Green in an article entitled “Optical and Retinal Factors AffectingVisual Resolution”, published in the Journal of Physiology, No. 181,page 576 to 593, (1965). Campbell and Green projected a brightly litextensive grid structure onto the retina and the image reflected by theeye was deflected with a beam splitter mirror out of the beam path andimaged with a sharp focus outside of the eye on an image plane (screen).The surface imaging of the grid structure was used after its reflectionby the retina, that is after passing twice through the eye, served forthe determination of the modulation transfer function of the eye. Thephotometric evaluation showed that the quality of the reflex image camevery close to the image quality seen by the eye.

The closed, static recording device used by Campbell employed anextremely high image illumination by photoflash with the eye in a fixedposition. Such a device is not suitable for recording weakly illuminateddynamic exterior images on the retina while the rapid natural eyemovements take place. For this purpose light sensitive rapid sensors arerequired together with a recording technique which sufficientlysuppresses extraneous light in the open beam path and which is alsocapable of recording images at least with the repetition frequency ofcostomary video standards.

CCD cameras which record all image dots in parallel with a fixedintegration time and serially scanning image recording systems withindividual detectors (photodiodes or photomultipliers) are suitable forthese purposes. Serial scanning involves sensing the image dots in time,one after the other. Both of these techniques are adapted to customaryvideo standards.

A basic advantage of using the CCD recording technique is the longintegration time in each image dot or pixel of, for example 20 mscompared to the short residence time in each pixel of only 40 ns duringscanning. However, the serial recording technique has a number of otheradvantages in connection with the recording of very weak, rapidlychanging light signals against a strong background noise as compared tothe parallel recording technique. These other advantages make up for theshort integration time.

These other advantages are:

a serial signal processing which makes possible a direct analog furtherprocessing of the image in real time;

an efficient suppression of scattered light by the small momentaryvisual field during scanning;

a high preamplification with low background noise by the employedavalanche photodiodes and photomultipliers;

a high signal dynamic which enhances the large variations of the picturebrightness on the retina;

an efficient analog background noise suppression, for example by thephase-lock-in detection or by signal correlation; and

a simple correction of imaging or recording errors.

With regard to the object of the invention, the critical advantage of aserial image scanning is the further possibility to combine such imagescanning with a time lag, synchronous, serial laser image projectioninto the eye.

Due to these advantages of the serial scanning, as compared to film andvideo recordings, the serial scanning is used since the early fifties(1950), primarily for the recording of microscope images. Threerecording methods can be applied with a serial scanning. A first“flying-spot” recording is achieved by an area illumination of theobject and a spot-type or pixel-type scanning with a photosensor(photoreceiver). The second method also referred to as “flying-spot”involves scanning the object with a point light source and surface areapick-up with a photosensor. The third method referred to as “confocalscanning” involves a spot illumination and a simultaneous spot pick-upwith a photosensor. The same scanning device can be used for spotillumination and spot pick-up.

In practicing the first two methods, either the light source or thesensor is rigidly mounted, whereby either the sensor or the source ismovable on the object. In the third method the light source and thereceiver (sensor) are together depicted (confocally) on the spot to bescanned. In this confocal method the light source and receiver are heldin a fixed position relative to each other.

In order to highlight the novel merits of the invention and itstechnical embodiments, the current status of the applications of imagerecordings and laser projections into the eye will now be explained inmore detail.

U.S. Pat. 4,213,678 (O. Pomerntzeff and R. H. Web) (September 1980)discloses for the first time the second type of the “flying-spot”recording technique with the aid of a scanned laser beam used as anillumination source and a rigid large area photomultiplier used as asensor or receiver for the pick-up or recording of the inner structureof the eye. These components are part of a scanning ophthalmoscope forexamining the fundus of the eye.

An article by R. H. Web, G. W. Hughes, and F. C. Delori entitled“Confocal Scanning Laser Ophthalmoscope”, published in “Applied Optics”,Vol. 26, No. 8, pages 1492 to 1499 (1987), describes a furtherdevelopment of the above technique to a confocal arrangement with thesimultaneous scanning of the laser beam and the receiver axis of thephotomultiplier.

In the apparatus of Web, Hughes, and Delore, the retina is scanned by alaser beam in a raster pattern. The laser beam illuminates the objectpixel-by-pixel (dot-by-dot) and line-by-line. The photosensor(photomultiplier) measures the respective reflected light and transformsthe measured value sequence into a video signal. A television monitorpresents the video signal eventually as an image. These three operationsteps take place in exact synchronism. While the laser beam scansline-for-line the eye background, the television signal issimultaneously assembled.

The laser beam passes first through a modulator by which theillumination intensity can be controlled in an open loop manner. Thehorizontal line deflection is generally performed by a rapidly rotatingpolygonal mirror while the vertical deflection is performed by aswinging mirror. The pivot point of the scanning motion is located inthe pupil plane of the eye. The light reflected by the eye background orrather scattered by the eye background is collected over the entirepupil opening and supplied to the photoreceiver or sensor through animaging optic. In this manner the beam deflection is again cancelled anda stationary light or beam bundle is obtained which is imaged orrecorded on a small detector surface.

Web, Hughes, and Delori recognized in the above mentioned article thepossibility of using a confocal imaging in an ophthalmoscope forprojecting artificial images with the aid of a laser projection into theeye. This possibility was described as follows: “The laser beam isdeflected by a fast (15-kHZ) horizontal scanner and a slow (60-Hz)vertical scanner to project a standard format TV raster on the retina.Modulation of the beam permits projection of graphics or even gray scalepictures in the raster. While the patient is seeing the TV pictureprojected on his/her retina, an image of the retina is displayed on a TVmonitor.”

The direct projection of modulated light stimuli and patterns is used inmodern laser scan ophthalmoscopes as, for example, are manufactured bythe firm “Rodenstock” of Munich. These ophthalmoscopes are primarilyused for line of sight analysis, video line of sight determinations, andcontrast sensitivity measurements, whereby respectively only one laserwavelength is used.

Further proposals for the direct image transmission into the eye bylasers are known from the following two publications.

European Patent Publication 0,473,343 B1 of Nov. 19, 1995 (SonyCorporation) discloses a “direct viewing picture image displayapparatus”. This direct viewing display apparatus employs substantiallyonly the known technical solutions as described above involving confocalimaging. These confocal imaging techniques have been realized in laserscanning ophthalmoscopes that are produced, for example by the firm ofRodenstock Instruments of Munich and such ophthalmoscopes are on themarket.

The technical solution for expanding the image transmission of but onecolor out of three colors, is described in claims 10 and 11 of EuropeanPatent Publication 0,473,343 B1. Such a technique has also been used inother laser displays for many years. The shifting of the depth positionof the images on the retina as described in claims 12 to 16 of saidpatent has been applied in the form of similar measures in existingequipment.

The separation of two beams by distinguishing polarizations as describedin claims 16 to 19 and shown in FIG. 6 of the above mentioned SonyPatent, in order to project the same image into both eyes, is basicallyunsuitable for a “true” three-dimensional image presentation, becausethese images do not have any perspective differences. Further, saidmethod does not permit any dynamic nor any individual adaptation to theeye alignment and thus it is difficult to realize this teaching inpractice.

European Patent Publication 0,562,742 A1 (Motorola, Inc.), published inAugust, 1993, describes a direct view image display (apparatus) referredto as “direct retinal scan display” which involves the direct imagetransmission onto the retina as in the above described patent to Sony,however, with the difference the projection is accomplished bydeflection through spectacles worn by a person.

The Motorola disclosure does not add new solution proposals to theexisting prior art. The direct mounting of the entire display on thehead of the user as defined in claim 4 or the deflection of the beampath of the projector through spectacles as defined in claim 5, has beenrealized in so-called “virtual reality” spectacles or in pilot helmetsequipped with a “head-up display”.

It is necessary to satisfy different optical requirements that must bemet by the laser beam deflection in order to successfully project animage onto the retina. Such laser beam deflection requires, in additionto the special construction of the beam guiding elements following thebeam deflection, a special spectacle glass vaulting. The Motoroladisclosure does not address solutions of these basic optical problems.

SUMMARY OF THE INVENTION

The invention suggests a serially operating recording and projectingapparatus which makes it possible to record the images produced on theretina of the human eye when naturally viewing the environment (orexterior) and to modify or enhance these recorded images by anelectronic image processing. The enhanced image is then projected backinto the eye by a laser beam projector for synchronous superpositiononto the original image, (referred to as retina reflex image). Accordingto the invention all three basic colors red, green and blue are detectedfor the recording and projected back onto the retina.

The present object is basically different from a confocal laser scanningophthalmoscope in which the retina is illuminated and imaged in the samescanning operation, because in the apparatus according to the inventionthe area-type retina reflex image of the exterior is scanned in a firstscanning cycle in accordance with the “flying-spot” method and in asecond timely separated scanning cycle the enhanced laser image isprojected onto the retina. In the third or next following scanning cyclethe retina reflex image is again recorded and in a fourth cycle theenhanced laser image is projected onto the retina and so on. Since thesecycles change rapidly, a continuous sequence is generated in the eye aswhen watching television or a movie. In this continuous sequence theenhanced laser image follows synchronously and congruently the originalimage independently of any eye movements.

The invention distinguishes itself from all prior art known to theapplicant relating to the direct laser projection into the eye and fromthe projection of extraneous images into a closed cyberspace spectacle.The invention also distinguishes itself from the projection ofadditional images into an open HUD spectacle. The distinguishingfeatures are the direct coupling of the projection with theinstantaneous exterior image content and the new technical embodimentsfor realism of such coupling.

The recording and image processing of the retina reflex image has becomepossible due to the rapid advance in the recording and processingtechnique of weak optical signals. The irradiation intensity to whichthe retina is exposed in a natural environment ranges from a very brightexternal illumination of 10⁻⁴ W cm⁻² to about 10⁻⁷ W cm⁻². In a room theillumination intensity generated by a weak interior lighting is withinthe range of 10⁻⁵ W cm⁻² to 10⁻⁶ W cm⁻², see for example an articleentitled “Safety With Lasers and Other Optical Sources” by D. Sliney andM. Wolbarsht, (1980). According to R. H. Web et al. in an article“Flying-Spot TV Ophthalmoscope” as published in “Applied Optics”, Vol.19, No. 17, pages 299 and following, (1980), a sensitivity of 2×10⁻⁵ Wcm⁻² has been achieved with a photon counting photomultiplier and spotscanning with lasers in accordance with a television standard whiletaking into account a signal to noise ratio of 5.

An increase in the sensitivity to 10⁻⁷ W cm⁻² can, for example, beachieved by an improved noise suppression or a decreased localresolution or by the use of a spiral scan instead of a TV raster scan. Araster scan offers a reduced scanning speed in the center of the visualfield and thus an increased time for integration in the image center.

According to the invention the surface area retina reflex image of theexterior picture is picked-up by a scanning photosensor. Thus, theinvention uses a flying-spot-type image recording technique of the abovementioned first kind. A time-lag synchronous laser image projection withthe aid of the same scanning device can be considered as a confocaltechnique in the sense of the common spot scanning projection of thelight source and the photosensor onto the retina. However, a confocalscanning recording technique is not involved because the function of thephotosensor and the function of the laser are exchanged relative to oneanother compared to their conventional use. According to the inventionthe time lag received signals serve for a time lag modulation of thelaser source. Contrary thereto in the conventional method the lasersource serves for illuminating while light signals are simultaneouslyreceived.

BRIEF DESCRIPTION OF THE DRAWINGS

The invention will be described in the following specification withreference to example embodiments, wherein the Figures supplement thedescription.

FIG. 1 is a schematic diagram of an example embodiment of the recordingand projecting apparatus (into the eye) according to present claims 1 to5 illustrating an embodiment wherein the imaging between the scanner andthe eye is performed by two concave reflecting surfaces of an auxiliarymirror HS and through the inner surface of the spectacles BG;

FIG. 2 shows a schematic block diagram of an example embodiment of therecording and projecting apparatus (into the eye) according to presentclaims 1 to 4 and 6, illustrating an embodiment wherein the imagingbetween the scanner and the eye is performed on the concave auxiliarymirror surface BG′ of an otherwise convex auxiliary mirror HS and theconcave inner surface of the spectacle glasses BG;

FIG. 3 is a block diagram of an example embodiment of the rigid beampath between the recording apparatus and the projecting apparatus withphotosensors (receivers) and laser modulators according to claim 14;

FIG. 4 is a block diagram of an example embodiment with a flexiblecoupling of the recording and projecting apparatus to a beam switch andscanning unit by means of flexible glass fibers according to claim 15;

FIG. 5 is a schematic illustration of an embodiment in which therecording and projecting apparatus for both eyes is housed in aspectacle frame;

FIG. 6 shows schematically an embodiment of the beam path in the scannerwhen recording the retina reflex image and subsequently projecting theprocessed image onto exterior objects by switching the horizontalscanning mirror by an angle of 90° according to claim 23;

FIG. 7 is a schematic illustration of the optoelectronic and electronicsubcomponents and their interconnections;

FIG. 8 shows schematically the waveforms of the timely scanning andlaser projection sequences; and

FIG. 9 illustrates schematically the microconstruction of the scannerintegrated into a spectacle frame coupled through glass fibers to aportable recording and projecting unit and provided with a wirelesstransmission to an image processing computer.

DETAILED DESCRIPTION OF PREFERRED EXAMPLE EMBODIMENTS AND OF THE BESTMODE OF THE INVENTION

The invention suggests according to claim 1, a scanning recordingapparatus for the serial recording of the low luminosity retina refleximage of external AW objects on the retina NH of the eye AA as shown inFIGS. 1 to 3.

The same recording and scanning apparatus is also used for projectingthe processed image with the aid of lasers and image modulators on thesame, but opposite light path onto the retina according to claim 2 asalso shown in FIGS. 1 to 3.

The central feature of the invention is a special spectacles carried byan observer as shown in FIGS. 5, 6 and 9. The glasses of the spectaclesBG serve as transmission elements for outside light and as an imagingsurface for the reflex image from the retina as backscattered throughthe eye. The reflex image or backscattered light is transmitted throughfurther imaging elements and a biaxial scanner HSS for the horizontaldeflection and a biaxial scanner VSS for the vertical deflection, to aphotosenor or receiver (FIGS. 1 to 4).

The beam path is simultaneously constructed in such a way that theextension of the side line from the detector through the spectaclesalways ends up in the absorbing layer of a radiation sink SS. Theextension of the sight line of the eye through the glasses, however,extends to the exterior AW (FIGS. 1 to 6).

The simplest method of beam splitting at the glasses BG of thespectacles involves the use of mirror glasses that are 50% transmittingand 50% reflecting. However, it is also possible to use activeelectronically controllable mirrors which switch over from a completetransmission to a complete reflection in both scanning cycles.

The eye AA images on the retina parallel or almost parallel beam bundlesof the exterior AW. The pivot point of the beam bundles for differentview angles to the environment is located in the pupil AP of the eye AA.

The invention uses a simultaneous recording in both eyes and aprojection into both eyes as shown in FIGS. 5 and 6. The invention isfurther uses a substantially identical beam path for the left and righteye. In case the user has ametropia including different powers ofrefraction in the left and right eye, the invention provides that eitherthe spectacle glasses give the respective refraction correction bydifferent vaultings (or curvatures) of the outer side and the inner sideof the glasses for the individual adaptation or that the user wearscontact lenses. For users with normal vision, the vaulting (orcurvature) of the inner surface and of the outer surface of thespectacle glasses BG are identical to each other.

In the same way, the light backscattered by each individual pixel (imagepoint) of the retina out of the eye is a substantially parallel beambundle which passes along the identical path as the incoming light,however in the opposite direction onto the inner side of the partiallyreflecting glasses BG of the spectacle. The curvature or vaulting ofthis surface is so formed that together with the eye lens a second imageof the retina pixel is generated in the intermediate plane ZE as seen inFIG. 1. An auxiliary mirror HS collimates the beams again and imagesthese beams so that the collimated beams pass through the common pivotpoint on the axis of the horizontal scanning mirror HSS (as on the otherside through the eye pupil). A vertical deflection is performed by asecond scanner mirror VSS.

The imaging out of one eye into one eye with the aid of the two scanningmirrors, with the auxiliary mirror and with the glass mirrors of thespectacles while simultaneously providing a free exterior vision throughthe glasses BG requires a relatively strong beam deflection. Thedeflection by 180° through two concave mirror surfaces partiallyeliminates any occurring imaging errors. The recording beam path for theimage recording and the reverse beam path for the image projection areotherwise identical to each other whereby image distortions in the eyeare substantially avoided.

However, in connection with spherical mirrors, remainder image faultsoccur due to their substantial imaging errors and in spite of therelatively small required deflection angles of smaller than ±10°.Therefore, high quality mirror systems such as concave parabolic mirrorsand elliptical mirrors as defined in claim 5 are provided for therecording out of the eye and for projecting back into the eye.

According to the features of claim 6, an efficient reduction of imagingerrors is achieved with the help of imaging on two concave glasses BGand BG′ of a surface HS which is otherwise convex in the oppositedirection. For this purpose it is possible to use the second half of theglasses of the spectacles as a full mirror surface BG′ having the sameconcave curvature as the glasses BG.

According to the invention any type of biaxial image scanners can beused, for example rotating mirrors or polygonal mirrors for the linedeflection and oscillating mirrors for the vertical deflection oracousto-optical deflection devices may be used for both axes.

In case a raster pattern scanning track is used with a separatehorizontal and vertical deflection, the image assembly can be made toconform with conventional video standards such as VHS, NTSC, and HDTV.

It is, however, possible to also use other scanning track patterns whichare better adapted to the image assembly of the eye than the rasterscan. For example, a spiral scanning track may be used. The largestvision sharpness of the retina is found in the area of the so-calledfovea centralis which takes up in the visual field only a small angularrange of about ±2° around the axis of vision. When a person directs itsattention to a fixed object, the eyes are normally moved in such a waythat the beams emanating from the fixed object fall onto the foveacentralis.

The residence time of the scanning beam in a spiral image scanningpattern increases continuously in the direction toward the visual axis.Therefore, a spiral scanning pattern is substantially better adapted tothe structure of the retina than a raster scanning pattern. Moreover,due to the increased duration of the residence time in the center, acorrespondingly higher signal to noise ratio is achieved in the centralarea.

Thus, for the above reasons the invention provides for using a spiralscanning pattern instead of a raster scanning pattern. The two beamdeflection units are respectively arranged and controlled for using aspiral scanning pattern.

The beam pass between the projection channel and the receiver orrecording channel is separated with the aid of a switching mirror SUS asin a laser scanning ophthalmoscope. Due to the good focussing of theprojection beam due to the small diameter of the laser beams, it ispossible to select the diameter of the projection beam to be smallerthan the diameter of the recording beam. Therefore, it is possible touse an aperture mirror for the separation of the two beam paths as shownin FIGS. 3 and 4. Due to the alternating use of the two beam paths it ispossible to achieve a more efficient method by using a flip-flop mirrorwhich switches the beam path synchronously with the scanning. The use ofa flip-flop mirror has the advantages of smaller optical losses in therecording channel and of a better optical screen out of any direct“cross talk” between the projection channel and the recording (receiver)channel.

As shown in FIG. 3, a focussing device FE is positioned in the beam pathof the laser projector unit upstream of the beam switch SUS foradjusting the size of the image spot projected by the laser beam ontothe retina. FIG. 3 also shows a visual field aperture GFB in thereceiver unit downstream of the beam switch SUS for varying the size ofthe scanning spot on the retina.

The visual field aperture GFB of claim 18 is positioned between twolenses and is provided in common for adjusting the momentary visualfield “seen” by the photomultipliers. The adjustment of the visual fieldaperture GFB is necessary for adaptation to the illumination conditionsat the retina and for adjusting the desired local resolution ordefinition. It is provided that both adjustments are accomplishedautomatically by controllers in response to computer control signals asshown in FIG. 7.

According to claim 12, it is provided that the retina reflex image isdivided by the use of dichroic filters DFR, DFG and DFB and by threeseparate detectors PMR, PMG and PMB into up to three color channels,whereby a substantially true color image can be recorded. On the laserside dichroic beam splitters DS are also used for collimating in acommon axis up to three laser beams in red, green and blue spectralrange (LR, LG, LB) following a separate image modulation of each colorMR, MG and MB.

For a true color image recording the optical signal is divided intothree basic colors by dichroic filters DFR, DFG and DFB in the recording(receiver) channel to provide the three color components upstream of thethree photosensors (receivers), preferably photomultipliers PMR, PMG andPMB. The three basic colors are separately measured. Due to the weaklight signals primarily photon counting methods should be used.

The invention further provides that the electronic image recorded by thedetector, is again converted, after image processing, into a serialoptical image with the aid of laser beam sources and modulators. Theconverted image is then, in a second imaging cycle, projected back intothe eye in synchronism with, but with a time lag relative to thescanning of the original retina reflex image. In the second imagingcycle the same optical components are used, however now functioning aslaser deflection unit (laser scanner) after reflection at the innersurfaces of the spectacle glasses BG.

The invention suggests in claim 20 to separate in time the periods ofimage recording from the periods of image projection, i.e. to performthese alternatingly as shown in FIG. 8. In this manner, the recording ofthe weak retina reflex image of the exterior AW is not disturbed by theprojection having a larger luminous intensity than the retina refleximage. For example, in a first image cycle the retina reflex image isrecorded and in a second cycle the processed electronic image isprojected into the eye. In a third cycle the retina reflex image isagain recorded and in a fourth cycle the processed image is againprojected back, and so forth.

When the image change takes place rapidly enough, the persistence of theeye makes sure that the two images appear for the viewer superimposed onone another, provided that the time lag of the picture projected intothe eye is shorter than the motion duration and persistence time of theeye and that the stability and resolution of the projected image iscomparable to the resolution of the eye.

The image repetition frequency must be sufficiently high in order torecord the unconscious rapid, interrupted motion of the eye having amean amplitude of 5 arc minutes and a duration between 10 and 20 ms andto record the rapid eye motions of 20° to 30° per second when the eyefollows a moving object over a larger angular range. The recording issubstantially adapted to the most rapid eye motions when a repetitionfrequency in the range of 50 Hz to 100 Hz is employed as is the case inthe television and computer techniques. This adaptation applies for theraster scanning as well as for the spiral scanning.

Further technical requirements to be met by the recording device relateto the size of the visual field encompassed by the eye and to the imageresolution of the present apparatus. For most applications the followingvisual field ranges are of interest. Namely, the area of highestdefinition vision having a diameter of 1° and about 7 million pixels inthe fovea centralis of the retina and the neighboring area having asubstantially lower definition up to about 10° diameter around the lineof vision. The spiral scanning pattern for the recording track isespecially suitable for satisfying these differing definition orresolution requirements.

Semiconductor lasers or miniaturized solid state lasers are provided aslight sources for the projection of images back into the eye. Theselasers have a low continuous wave power smaller than 300 μW which cannotcause any harm to the eye. In connection with the use of semiconductorlasers, the image modulation could be performed directly through thepower supply for the lasers. In order to produce all colors, it isrecommended that three lasers be used for the base colors red, green andblue. As shown by the known color triangle of the human vision, it ispossible to generate all other colors including the non-colors gray andwhite by color summation of monochromatic laser lines of these basecolors. The invention also includes the possibility of using anyindividual color in a monochromatic embodiment. As shown in FIG. 7, theinvention provides a signal processor SP which processes electronicallythe direct image from the retina and which coordinates synchronously allfunctions of the apparatus, including the functions of the scanners VSSand HSS, of the laser spot adjustment and of the size of the viewingfield aperture LAA/GFB. The image processing computer BVC receives theimage or images observed by the eye from other technical sensorssupplying their signals to an external terminal EA to the computer BVCwhich processes these images in accordance with a given program SW. Suchprocessing taking place prior to modulating these images in the form ofimage signals onto the laser beams with the aid of the signal processorSP.

The laser projection makes it possible to superimpose synchroneously anexterior AW image onto the retina reflex image perceived by the eye. Forthis purpose the image processing computer BVC processes not only theperceived retina image for projection back into the eye for fading intothe perceived image, but it also processes the exterior images which aresupplied to the computer through a respective input EA. If the timeduration between image recordal and projection is respectively shortcompared to the rapid eye movements, the eye will no longer perceive anyimage interruption, just as when watching a television screen.

The separate, but simultaneous image scanning from both eyes alsorecords the perspective differences of both images. A reconstruction ofthe three-dimensional vision is assured because this three-dimensionalvision is included in the laser return projection into both eyes.

In addition to the projection of the retina reflex images after imageprocessing back into the eye, the apparatus according to claim 25 alsomakes possible the projection of these laser images directly onto theobject seen by the eye in the exterior AW. This embodiment of theinvention is shown schematically in FIG. 6 and is achieved by flippingthe scanning mirror through an angle of 90°.

The structural elements used according to the invention are todaysubstantially miniaturized and are available at reasonable costs. Thebeam deflection unit and the scanner can be installed in a simplespectacle frame B as shown in FIG. 9. By using a glass fiber conductorGFL it is possible to house the laser projector unit and the recordingunit in a separate small housing TOE, for example having the size of apaperback book including a power supply battery. The data exchange withan external fixedly installed image processing computer can beaccomplished either by a radio wave link or by an infrared radiationlink. All elements of the apparatus according to the invention can thus,in the light of the state of the art, be carried by a person withouttrouble. Further, the wireless image data exchange with an externalcomputer makes possible an unlimited freedom of motion.

The various areas of use of the invention can be summarized in thefollowing four categories:

recording of images of the exterior AW, the processing and reverseprojection for fading the back-projected image into the original imagein the eye;

superimposing of images of other recording systems, for example of thesame scene in other spectral ranges onto the original image;

superimposing of virtual images generated solely by the computer; and

recording of images of the exterior AW and projecting these images notinto the eye, but rather back onto the same objects that are seen by theeye in the exterior.

The first category involves applications with the aim of improving theimage seen by the eye, by a purposeful image summation, for example toenhance the resolution and to amplify an unclear or weakly illuminatedimage which is of great help to people with normal vision as well as forpeople with a defective vision.

Other possibilities of modifying an image would, for example, involvethe change of the color of objects by a new color summation. Thistechnique could be used for a purposeful white coloring of certain areaswithin the vision field and thus to erase or reduce the opticalinformation.

The second category involves the superposition of images of the samescene, for example taken in the infrared range (invisible) or seen byradar devices. This technique would substantially facilitate, forexample, the driving of a vehicle or flying an aircraft at night andunder foggy or hazy conditions.

For medical applications, it is possible to superimpose, for exampleX-ray images, acoustical images or images obtained by magnetic resonancetomography, onto the direct image of the patient's body or the patient'sorgans for facilitating the diagnosis made by the physician or forpurposes of surgery.

The third category involves applications in which the image is enhancedby fading in virtual auxiliary images, e.g. when using the conventionalHUD devices while driving a vehicle. The invention provides theadditional advantage of an exact synchronization of the fading-in withthe exterior image seen by the eye. In this manner it would be possibleto fade-in extraneous images with small image content or as a stereoimage at a different distance than the other objects.

Interactive applications from the computer technology belong into thisthird category, namely the fading-in of a virtual computer mouse(cross-hair) which can be guided alone by eye movements (instead byhand) over real objects in the exterior (also display). In this context,the initial click-on or an order could be performed by additional eyemovements such as an eyelid flip or by voice or by pushing a button.

The third category also includes applications in cyberspace, i.e. thefading of virtual computer full images into the closed spectacles.According to the invention it is possible to use a recorded retinareflex image of the fade-in virtual images in order to stabilize theseimages relative to eye movements.

The fourth category describes a type of “active viewing”, i.e. a sceneseen by the eye and recorded by the scanner is serially illuminated inthe next scanning cycle by a laser image spotlight or projector. Thethus illuminated scene is again recorded by the eye and leads in thenext following cycle to a changed second laser illumination which isfollowed by a third cycle and so forth. In this manner an opticalfeedback loop is generated which, depending on the respectivearrangement of the illumination, can either be used as a positivefeedback or a negative feedback for the various applications, forexample in order to brighten objects that are hard to recognize or toincrease their contrast, or to change their color.

What is claimed is:
 1. An apparatus for recording a retina (NH) refleximage of the exterior as perceived by the human eye, comprising anoptical transmitter/reflector for transmitting rays of an exterior imageinto the human eye and for reflecting said retina reflex image, anoptoelectronic receiver/detector for serially recording said retinareflex image, wherein said optical transmitter/reflector comprises abeam splitter mirror or a controllable mirror (BG) having a vaulted eyefacing surface, said apparatus further comprising a biaxial scanner(HSS; VSS) positioned for scanning said retina (NH) reflex image asreflected by said beam splitter mirror, said biaxial scanner deflectingsaid retina reflex image along a reflex image beam path to saidoptoelectronic receiver/detector for said serial recording of saidretina reflex image.
 2. The apparatus of claim 1, further comprising alaser beam projector for projecting and superimposing a laser image ontosaid retina reflex image.
 3. The apparatus of claim 2, furthercomprising an image processor for processing said retina reflex image toproduce a modified image, and wherein said laser beam projector projectssaid modified image back into said human eye.
 4. The apparatus of claim2, further comprising an image input device for providing to said laserbeam projector further images for projection by said laser beamprojector into said human eye by laser beams.
 5. The apparatus of claim2, further comprising a laser beam path deflector for deflecting a laserbeam of said laser beam reflector onto objects seen by said human eye.6. The apparatus of claim 2, further comprising image modulatorspositioned in a beam path of said laser beam projector for generating onsaid retina (NH) said laser image synchronously with said scanning ofsaid retina reflex image but with a time lag to said scanning.
 7. Theapparatus of claim 2, further comprising a device (SUS) positioned in areflex image beam path of said retina reflex image, for projecting saidlaser image back into said eye along said reflex image beam path, but inthe opposite direction relative to the direction of said retina refleximage.
 8. The apparatus of claim 2, further comprising an illuminationchannel, and beam splitters in said illumination channel for collimatinga plurality of laser beams into said reflex image beam path forprojection onto said retina (NH).
 9. The apparatus of claim 2, furthercomprising a flexible fiber optical conductor (GFL) for coupling saidoptoelectronic receiver detector and/or said laser beam projector tosaid biaxial scanner (HSS, VSS).
 10. The apparatus of claim 2, furthercomprising a beam focussing device positioned for focussing said laserbeam of said laser beam projector for varying the size of an image spotof said laser image on said retina (NH).
 11. The apparatus of claim 1,further comprising an image processor computer (BVC) for processing saidretina reflex image in synchronism with said scanning of said retinareflex image, said image processor computer (BVC) being further capableof synchronizing extraneous images with said retina reflex images or ofsynchronizing computer processed informations with said retina refleximages.
 12. The apparatus of claim 1, further comprising in said refleximage beam path forming an optical receiver channel, a plurality of beamsplitters (DS) and a respective plurality of photodetectors (PMR, PMG,PMB) for detecting a plurality of spectral ranges independently of eachother.
 13. The apparatus of claim 1, further comprising spectacles, andwherein said apparatus is arranged in said spectacles.
 14. The apparatusof claim 13, wherein said spectacles comprises glasses forming said beamsplitter mirror (BG).
 15. The apparatus of claim 1, further comprising avariable visual field aperture (GFB) in said reflex image beam path ofsaid optoelectronic receiver/detector for varying the size of a scanningspot on said retina (NH).
 16. A method for recording a retina (NH)reflex image of the exterior as perceived by the human eye (AA),comprising the following steps: (a) scanning said retina reflex imagefrom said retina (NH) to provide a scanned image, (b) deflecting saidscanned image into an optoelectronic receiver/detector, and (c) seriallyrecording said scanned image to provide a recorded scanned image. 17.The method of claim 16, further comprising (d) projecting said recordedscanned image or an extraneous image back into said human eye (AA), and(e) synchronously superimposing said scanned image or said extraneousimage onto said retina reflex image, whereby said scanned image may be amodified image or an enhanced image.
 18. The method of claim 17, furthercomprising alternating in time said scanning step and said projectingstep so that a sequence of scanning steps fits into time slots betweenprojecting steps in a sequence of projecting steps and vice versa. 19.The method of claim 16, wherein said scanning step (a) is performedalong a raster scanning pattern or a spiral scanning pattern.
 20. Themethod of claim 16, further comprising modifying or enhancing saidretina reflex image by any one or more of the following image processingsteps: image summation, color variation or modification, superimposingof images of the same scene, but taken in spectral ranges outside thevisible range, and fading-in virtual images.
 21. The method of claim 20,comprising fading-in a virtual computer mouse, and guiding said virtualcomputer mouse solely by eye movements for performing instructions inresponse to said eye movements.
 22. The method of claim 20, furthercomprising fading-in virtual computer generated images and using thevirtual computer image received by said retina for stabilizing saidvirtual computer images.
 23. The method of claim 16, further comprisingserially illuminating by a laser image spotlight a scene seen by saideye and recorded on said retina (NH).